Skip to content

fix(server): prevent unrelated sandbox deletes from blocking deletion events#2340

Draft
pimlock wants to merge 2 commits into
mainfrom
2326-unblock-sandbox-delete-events/pimlock
Draft

fix(server): prevent unrelated sandbox deletes from blocking deletion events#2340
pimlock wants to merge 2 commits into
mainfrom
2326-unblock-sandbox-delete-events/pimlock

Conversation

@pimlock

@pimlock pimlock commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

🏗️ build-from-issue-agent

Summary

Prevent one slow sandbox deletion from blocking unrelated watcher events or deletes. The gateway now serializes duplicate deletes per stable sandbox ID, persists and recovers deletion transitions with version checks, and performs backend calls outside the gateway-wide state guard.

Related Issue

Closes #2326

Changes

  • Add weak, per-sandbox delete gates so duplicate requests share one driver call without blocking other sandbox IDs.
  • Bind a delete to the ID resolved at request start, preserve the workflow after client cancellation, and never retarget a reused sandbox name.
  • Make watcher reconciliation deletion-aware and ignore unknown backend snapshots while logging the dropped event.
  • Recover failed driver deletes from an observed backend snapshot, remove an absent backend row, or roll back the exact deleting version when the backend state is inconclusive.
  • Delete sandbox rows by stable ID and resource version while retaining the existing best-effort name-based settings cleanup.
  • Extend concurrency, cancellation, name-reuse, cleanup, and recovery regression coverage; harden lifecycle E2E polling and diagnostics.
  • Document the stable deletion invariants and the current gateway-restart limitation.

Deviations from Plan

Settings cleanup intentionally retains its existing best-effort name-based behavior. An earlier atomic persistence extension was removed to keep this PR scoped to the deletion concurrency issue.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (execution delegated to the labeled Branch E2E Checks workflow)

Tests added or updated:

  • Compute: 25 regression tests covering lock isolation, duplicate deletes, cancellation, name reuse, watcher races, recovery, and cleanup.
  • E2E: 2 sandbox lifecycle cases now use deadline-based polling with final-state diagnostics.

Checklist

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock pimlock added the test:e2e Requires end-to-end coverage label Jul 17, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/2340 is at {"messa while the PR head is 0f18d50. A maintainer needs to comment /ok to test 0f18d50c78cdb7ae46861508c6641342028ad665 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

@pimlock

pimlock commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 0f18d50

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 8283eec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(server): prevent unrelated sandbox deletes from blocking deletion events

1 participant